Information entropy Totally Explained

Everything about Shannon Entropy totally explained

In information theory, the Shannon entropy or information entropy is a measure of the uncertainty associated with a random variable. It quantifies the information contained in a message, usually in bits or bits/symbol. It is the minimum message length necessary to communicate information.
   This also represents an absolute limit on the best possible lossless compression of any communication: treating a message as a series of symbols, the shortest possible representation to transmit the message is the Shannon entropy in bits/symbol multiplied by the number of symbols in the original message.
   A fair coin has an entropy of one bit. However, if the coin isn't fair, then the uncertainty is lower (if asked to bet on the next outcome, we'd bet preferentially on the most frequent result), and thus the Shannon entropy is lower. A long string of repeating characters has an entropy of 0, since every character is predictable. The entropy of English text is between 1.0 and 1.5 bits per letter, or as low as 0.6 to 1.3 bits per letter, according to estimates by Shannon based on human experiments. Equivalently, the Shannon entropy is a measure of the average information content the recipient is missing when he does not know the value of the random variable.
   The concept was introduced by Claude E. Shannon in his 1948 paper "A Mathematical Theory of Communication".

Definition

The information entropy of a discrete random variable X, that can take on possible values ,dx.

The relative entropy carries over directly from discrete to continuous distributions, and is invariant under co-ordinate reparametrisations.

Further Information

Get more info on 'Shannon Entropy'.

External Link Exchanges

Do you know how hard it is to get a link from a large encyclopaedia? Well we're different and will prove it. To get a link from us just add the following HTML to your site on a relevant page:

<a href="http://information_entropy.totallyexplained.com">Information entropy Totally Explained</a>

Then simply click through this link from your web page. Our crawlers will verify your link, extract the title of your web page and instantly add a link back to it. If you like you can remove the words Totally Explained and embed the link in article text.
As long as your link remains in place, we'll keep our link to you right here. Please play fair - our crawlers are watching. Your site must be closely related to this one's topic. Any kind of spamming, dubious practises or removing the link will result in your link from us being dropped and, potentially, your whole site being banned.